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HIGHLIGHTS 


► Combination of color contrast enhancement and reference palette techniques. 
^ Missing values are interpolated using the nearest Euclidean distance. 

^ The method is validated through objective metrics. 
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The increasing availability and deployment of imaging sensors operating in multiple spectral bands has 
led to a large research effort in image fusion, bringing a lot of pixel-level color fusion algorithms. Color 
constancy and color contrast are two important topics for color image fusion. However, there is no work 
considering these two aspects in the meanwhile. The paper focuses on enhancing the color contrast 
between targets and their backgrounds and maintaining the color constancy. Firstly, a group of visible 
and infrared images are rendered with natural colors with color contrast enhancement algorithm. Then 
a color reference palette is derived from source images and their corresponding fused results. To enrich 
this palette, the nearest Euclidean distance determines the absent colors. The complete palette can be 
deployed in real-time to different multi-band image sequences of similar scenes. Experimental results 
based on different data sets show that the targets are popped out with intense colors while backgrounds 
present natural color appearance. Some objective analysis also shows that the presented method not only 
inherits more detail information from source images but also brings bigger color distance when compar¬ 
ing with the existing algorithms. 

© 2012 Elsevier B.V. All rights reserved. 


1. Introduction 

Image fusion is a methodology concerned with the integration 
of multiple images into a composite image. It will benefit the im¬ 
age analysis in many fields, such as studies in remote sensing, ma¬ 
chine vision, and molecular biology [1]. 

Until recently a gray or green-scale representation of nightvi- 
sion imagery has been the standard. However, the increasing avail¬ 
ability of multispectral imagers and sensor systems has led to a 
growing interest in color display of night vision imagery. In princi¬ 
ple, people can discriminate several thousands of colors but only 
about 100 shades of gray at any instant. As a result, grayscale 
nightvision images are usually hard to interpret and may give rise 
to visual illusions and loss of situational awareness. However color 
fused images may improve feature contrast which may bring faster 
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and more accurate object identification, scene understanding and 
reaction time. Also, observers are able to selectively attend to 
task-relevant color targets and to ignore non-targets with a task- 
irrelevant color. Thus, techniques for fusing IR and intensified vi¬ 
sual imagery should be employed in order to provide a compact 
representation of the scene with increased interpretation capabil¬ 
ities [2,3]. A number of techniques have been proposed to render 
night-time imagery in color [4-8]. A simple mapping of multi¬ 
bands into the three components of an RGB or YUV image can pro¬ 
vide an immediate benefit. On the other hand, inconsistent color 
mappings may hinder situational awareness due to lack of physical 
color constancy [8]. Thus, an image fusion method for night vision 
imagery must result in color images with natural and stable 
appearance and a high degree of similarity with the corresponding 
natural scenes. 

In 2003, Toet originally applied global color transferring [9] to 
render night-time imagery with a natural color appearance 
[10,11]. This method matches the first order statistical properties 
of night-vision imagery to those of a color reference image. As long 
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as they are similar to each other to some degree, satisfactory 
results can be obtained. When the composition of the reference im¬ 
age is different significantly from that of the nightvision image, the 
result may look unnatural. To overcome this shortcoming and get 
color constancy, Hogervorst and Toet [3,12] presented a lookup-ta¬ 
ble technique to optimize the match between the multi-band im¬ 
age and the reference image. Once it has been derived the color 
mapping can be deployed in real-time to different multi-band im¬ 
age sequences of similar scenes. Although the advantage of Toet’s 
method for situation awareness in real-time application task has 
been well demonstrated, its effect on target detection is still weak 
as their methods process all three channel of the color space with 
the same linear mapping in the color transfer step, resulting in low 
contrast between the target and the background. So targets that 
are clear in the IR image sometimes may become invisible and 
are difficult to detect, that may hinder target detection tasks 
[13]. Thus, ideal image fusion should not only improve the situa¬ 
tion awareness performance but maintain or increase the high 
detectability of the target in the IR image [14-16]. 

Image fusion schemes based on image analysis and nonlinear 
color mapping can easily meet this demand [8,17,18]. Zheng and 
Ma [8,18] presented a new “local-coloring” method that functions 
to render the nightvision image segment-by-segment by taking 
advantage of image segmentation, pattern recognition, histogram 
matching, clustering and merging techniques. Instead of extracting 
the color from a single reference image, the statistical properties 
are from a set of natural scene images. Depending on the scene 
contents and colors, the targets in the rendering results can dem¬ 
onstrate distinguishable colors while the whole scene possesses 
natural and realistic coloration. However, regarding computational 
complexity, the local-coloring method is even more expensive than 
Toet’s original color transform, since it involves time-consuming 
procedures such as nonlinear diffusion, histogram analysis and im¬ 
age segmentation. Simpler color contrast schemes were obtained 
by using linear fusion and nonlinear color transferring [13]. Yin 
and Qian [13,19] introduced a ratio of local to global divergence 
into color transferring equation. The hot and cool targets both 
can be popped out with intense colors. However, just like Toet’s 
original method, the reference image must have the similar com¬ 
position with the false fused image. 

Benefitting from Toet’s and Qian’s methods, here we will intro¬ 
duce a new simple and fast method to consistently apply natural 
daytime colors to multi-band nightvision imagery when bringing 
color contrast between the targets and the background. The new 
method is illustrated in Fig. 1. First, we generate a series of color 
fused samples using color contrast enhancement approach in 
YUV color space. Next, a reference color palette is produced from 
corresponding the source multi-band images and their correspond¬ 
ing fused results. To make it fulfilled, we fill in the absent colors by 
calculating the nearest Euclidean distance. Once the palette is de- 



Fig. 1 . A typical color image fusion system. 


rived the color mapping can be easily deployed in a real-time 
application. 

2. Samples based color contrast enhancement 

Since combining simple linear false color fusion and nonlinear 
color transferring can bring good color contrast between the target 
and the background, we still adopt a similar scheme to produce a 
group of color fused samples which will be the color sources of 
the reference palette. Different from Qian’s method, the multi¬ 
band sensor images are rendered directly in YUV color space. 
YUV space is commonly used in video transmission and processing 
system. Y denotes luminance, U stands for difference between blue 
and luminance, and V represents difference between red and lumi¬ 
nance. Image fusion and color transfer in this space will need not 
iterative color space transformation. Firstly, each pair of visible 
and infrared images is transformed to a false-color image by taking 
them as input to the three channels of a YUV image. The false color 
fusion is carried out as: 

(Y = l vis 

l U = I vis -l IR (1) 

{v = i m -i vis 

where I vis and I IR stand for the visible and IR images, respectively. By 
mapping the visible image to the achromatic channel, high-fre¬ 
quency information is kept in the fused image. The differencing 
operation between the visible image and IR image provides inter¬ 
sensor contrast. The resulting image agrees with our natural associ¬ 
ations of warm (red) and cool (cyan). 

The false color fused images have unnatural color appearance 
and lack of color constancy. In the following step, linear and non¬ 
linear color transfer is implemented to render the false color fused 
images with natural color appearance and enhance the color con¬ 
trast between the targets and the backgrounds. In Yin’s and Qian’s 
methods some information features (like edges and corners) that 
are available in the multi-band images may disappear. Especially, 
in Qian’s method, multi-band images are pre-processed by median 
filtering and then are fused by a weighted method that may both 
result in weak edges that are apparent in the source images. To 
maintain these high-frequency information, a strengthening expo¬ 
nent p is introduced into transferring equation in V channel which 
will further enlarge the color distance between different details. 
Color transferring equations are described as: 

, Y = ^(y I -f4) + i4 

< U'=^(U f -n]!) + H U y ( 2 ) 

V'= W - a ^(V f -nJ) + n v y 

where w is defined by luminance distance in IR image between each 
pixel IR{iJ) and the average luminance value ji IR as: 

w(i,j) = \\IR(i,j)-n IR \\e/(dist) (3) 

M N 

(dist) = - A*«lI/M x N w 

i =1 j = 1 

As both hot and cold targets are of great divergences from the 
mean intensity of the IR image, both of them can be enhanced. 
With color contrast enhancement, hot targets will appear intense 
red, while cold targets will appear cyan in the fused images. The 
red color of hot targets can alert observers of possible danger, 
and the cyan color can help observers to detect cold targets or 
low temperature regions [13]. 
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(a) 




Fig. 2. Reference palette from NV4917 pairs: (a) non-amended palette (b) amended palette. 



Fig. 3. Color mapping based on the palette, (a) Color mapping with Fig. 2a. (b) Color mapping with Fig. 2b. 


3. Color mapping based on reference palette 

As the statistical approach inherently only addresses the global 
color characteristics of the depicted scene, there is no strict rela¬ 
tionship between sensor output values and object color. Similar 
to lookup tables, here we will describe a fixed color transformation 
that makes object colors determined by multi-band sensor values. 
In this approach color mapping is derived from a subset (or all pos¬ 
sible) multi-band sensor outputs. This (sub)set of multi-band sen¬ 
sor outputs forms the reference set. The color attributed to any 
multi-band sensor value is equal to the color corresponding to 
the reference sensor output that is most similar to the actual sen¬ 
sor output. To achieve this, a reference palette is built based on a 
set of samples for which both the reference sensor outputs and 
their corresponding natural fused colors (YUV-value) are known. 
It can be defined as a matrix P: 

PM = mi) G(U) B(U )) (5) 

where i and j separately denote the visible and infrared sensors out¬ 
put values changing from 0 to 255. R(i, j), G(i, j) and B(i, j) are final 
natural rendered colors. 

To obtain the final natural colors with coordinate (i, j), first, the 
pixels with gray values i and j are located in each set of visible and 
infrared images separately. Then finding all corresponding pixels in 
their exactly corresponding natural fused results. Next, we 


calculate the average YUV-vector over this ensemble of pixels. Fi¬ 
nally the resulting average values are transformed back to RGB. 
This process yields the matrix containing the natural colors associ¬ 
ated with the distribution of the pixels: colors that appear more 
frequently are attributed a greater weight. 

Unfortunately, we cannot find enough multi-band images to 
cover all possible 256 x 256 combinations, so the rendered re¬ 
sults tend to include some undesired black points when applying 
the reference matrix to the infrared and visible images with sim¬ 
ilar scenes. To promote the reference matrix, we calculate the 
Euclidean distance between the absent combinations and the ex¬ 
isted multi-band sensor values. And the nearest match deter¬ 
mines the absent RGB-values. Once the full reference matrix 
has been defined, it can be used in a real-time application. 
And the color of objects depends only on the multi-band sensor 
output values and is independent of the rest of the image 
content. 

4. Experimental validation 

4.1. Visual assessment 

To validate the effectiveness of our method, we select two 
groups of multi-band image sequences provided by Alexander Toet 
and TNO Human Factors Research Institute and compare 
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subjectively our fused results with those brought by Toet’s method 
[10] without enhancement and by Yin’s and Qian’s enhancement 
methods [13,19]. 

In the first experiment, we randomly select 16 pairs from 
NV4917 to produce reference palette and the rest three pairs are 
tested using the generated palette. Figs. 2 and 3 show the reference 
palettes generated from 16 pairs and the corresponding mapping 
results. Because 16 pairs cannot cover all gray levels, the palette 
presents many black points (seen in Fig. 2a); When we directly 
map it to three pairs of multi-band images, hot target cannot be 
rendered effectively which can be seen in Fig. 3a. To overcome this 
shortcoming, we adopt the minimum Euclidean distance to com¬ 
pensate the absent points. Fig. 2b gives the amended palette and 
Fig. 3b well show the overall rendered effects. 

Fig. 4 demonstrates the comparative fused results. Fig. 4a are col¬ 
orizing results without color contrast enhancement. Fig. 4b and c are 
corresponding enhanced results brought by Qian’s and Yin’s meth¬ 
ods separately. Obviously, Qian’s, Yin’s and our rendering effects 
can all pop out hot targets. The man appears intense red or yellow 1 . 
When comparing with Qian’s method, tests show that details such 
as brooks in the foreground are clearer and more apparent in Yin’s 
and our results, while part of them is covered by grass in Qian’s 
method. It is because that in the equation of false fusion, the IR 
and visible images are both mapped to the achromatic channel 
by a weighted method. That can get bigger targets’ area but it is 
at the cost of losing details. In addition, the overall color is milder 
in our rendered results. While some areas in Qian’s effects are very 
dazzling which probably reduces the comfortable level of observa¬ 
tion. When comparing with Yin’s fused results, the man become 
bigger and more integrated. In addition, the details are clearer 


1 For interpretation of color in Fig. 4, the reader is referred to the web version of 
this article. 


and the backgrounds can present more natural colors in our result. 

The same conclusions can be deduced from experiments using 
NV7404 pairs. In this testing, 17 pairs are used to produce refer¬ 
ence palette. Using three remaining pairs, Fig. 5 gives comparing 
effects brought by different methods. Without channel’s enhance¬ 
ment the man is hard to distinguish from the background (e.g. 
Fig. 5a). Other three methods all provide better results (seen in 
Fig. 5b to d). The man appears red and intense orange-yellow, thus 
can be easily distinguished from the backgrounds. In addition, our 
method (Fig. 4d) not only pops out the target but also brings more 
natural and clearer backgrounds. 


4.2. Objective assessment 

In many applications, the end user of the fusion result is a hu¬ 
man, thus, the human perception of the composite image is of par¬ 
amount importance and therefore, fusion results are mostly 
evaluated by subjective criteria. Objective performance assessment 
is a difficult issue due to variety of different application require¬ 
ments and the lack of a clearly defined ground-truth [14,20]. For¬ 
tunately, there have been some effective performance measures 
according to specific tasks such as the percentage of correct classi¬ 
fications, false-alarm rate and color distance. Here, we mainly aim 
at improving the identification ability of the targets by enhancing 
the color contrast between targets and their surrounding back¬ 
grounds, so we are interested in evaluating different color fusion 
methods through color distance. In addition, we are also interested 
in measures that express the successfulness of fusion techniques 
that create a composite image that retains salient information from 
the source images while minimizing the number of artifacts or the 
amount of distortion that could interfere with interpretation. The 
following will gives these measurements. 
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Fig. 5. Fused results of different methods, (a) Fused results without enhancement, (b) Qian’s fused results, (c) Yin’s fused results, (d) Our rendering results. 


The hit rate of color vision in detecting a target relies mainly on 
its ability to characterize color [21 ]. When the target is far from the 
background in color, it improves the object-background discrimi¬ 
nation. So we measure the color distance between the target region 
and its surrounding region. Since image signals are generally 
non-stationary, it is appropriate to measure the color distance be¬ 
tween the target region and its surrounding region over local re¬ 
gions and then combine the different results into a single 
measure. We use a sliding window of fixed size which moves from 
the top-left corner to the bottom-right corner. For each window w, 
the color distance is computed by: 


W,B) 


(Jab 2 AB 2 o A o B 

OaO B ' (A ) 2 + (B ) 2 ' o\ + o\ 


( 8 ) 


where o A and o B are standard variance of images A and B , respec¬ 
tively. a AB is their covariance. 


o A = 


(Tab = 


MN 


tr EL,-V 

-i M N _ 

-Y;^(A(m,n)-A)(B(m,n)-B) 


MN 


( 9 ) 

( 10 ) 


D(t, b\w) = yJ{(Y t ) - (n)) 2 + m) - <U„)) 2 + «V t ) - (Vs)) 2 (6) 

where t , b stand for the target and the background regions, respec¬ 
tively. () is the mean value. 

And the overall image color distance is defined by averaging all 
local distances: 

D(t,i)=iEC(t,%) (7) 

I I weW 


In formula (8), the first component is the correlation coefficient 
between images A and B , which measures the degree of linear cor¬ 
relation between these two images. The second factor corresponds 
to a kind of average luminance distortion. The third component 
measures a contrast distortion. The bigger this value is, the more 
similar the two images are. As the rendered results are from infra¬ 
red and visible images, we need to consider the fusion quality of 
two source images with fused result. Here the relation function 
of them is reconstituted as: 


where W is the family of all windows and \ W\ is the cardinality of W. 

Color distance only evaluates rendering results themselves, and 
cannot assess how well the results retain the information of the 
source images. Piella [20] introduced an effective method to mea¬ 
sure the similarity of two images. The formula is as following: 


Q(A,B,F) = AQ(A,F) + (1 - 2)Q(B,F) (11) 

where A, B and F respectively stand for visible, infrared and fused 
images. A e [0,1] is a weight coefficient, which weighs the contribu¬ 
tion of two source images to the fused result. 
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Table 1 

Color distance (s). 



The first group 


The second group 



1 

2 

3 

1 

2 

3 

Dyin 
^3 Qian 

Dour 

160.7399 

197.6560 

201.8396 

180.4722 

186.4178 

203.4562 

178.2415 

178.7850 

205.1244 

146.1892 

167.8713 

190.1020 

136.0798 

154.9974 

184.1719 

149.8006 

169.0587 

190.9615 


Table 2 

Inheriting information (s). 



The first group 


The second group 


1 

2 

3 

1 

2 

3 

QYin 

0.5241 

0.5095 

0.4826 

0.5108 

0.5085 

0.5086 

QjQian 

0.4981 

0.4762 

0.4456 

0.4793 

0.4759 

0.4735 

Qour 

0.5341 

0.5199 

0.5041 

0.5221 

0.5181 

0.5180 


Table 3 

Running time (s). 


Toet’s (2003) 

Yin’s 

Qian’s 

Our method 

The first group 

0.9612 

0.7917 

1.0540 

0.6102 

The second group 

0.9467 

0.8141 

0.9802 

0.6123 


Table 1 shows the comparison of color distance among three 
enhancement methods. Qian’s and our methods bring bigger color 
distance due to the strengthening factor that further enlarges the 
distance of different details to the mean color value. In addition, 
the intense red colors and more natural backgrounds brought by 
our method make the color distance improve about 40% than the 
Yin’s. 

Table 2 gives the comparison of inheriting information from the 
source images. It can be seen that Qian’s method shows the worst 
performance because some details are lost by the weighted meth¬ 
od. While Yin’s and our methods can relatively maintain these 
information well. 

The effectiveness of the proposed method can be further ex¬ 
plained through the running time. Table 3 gives the average run¬ 
ning time of the three fused results for different methods on a 
Intel (R) CoreTM 2 Duo 1.50 GHz CPU and 1 GB memory using 
MATLAB code. It can be easily seen that the proposed method is 
the fastest. Because it only need simple color mapping and avoid 
false fusing and color transferring once the reference palette is 
produced. 

5. Conclusions 

An improved color image-fusion scheme is presented by com¬ 
bining a color contrast enhancement approach and a palette tech¬ 
nique. Color contrast enhancement approach helps to produce a 
series of color fusion results, where hot targets will appear intense 
red and cold targets will appear cyan under the control of an 
enhancement ratio in accordance with the characteristics of the 


target and the background in IR image. Then the palette is gener¬ 
ated from a set of multi-band night-time images and their corre¬ 
sponding fused results. To obtain a complete palette, the nearest 
Euclidean distance determines absent colors. Once it is established, 
the method allows for a real-time software implementation that 
requires a minimal amount of computer processing. 
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